Skip to content

[Experimental] Build the lectures with QuantEcon/mystmd instead of jupyter-book - #363

Open
kp992 wants to merge 55 commits into
mainfrom
jb2
Open

[Experimental] Build the lectures with QuantEcon/mystmd instead of jupyter-book#363
kp992 wants to merge 55 commits into
mainfrom
jb2

Conversation

@kp992

@kp992 kp992 commented Jul 23, 2025

Copy link
Copy Markdown

Experimental setup for building the lectures with the QuantEcon MyST stack. This PR is built on top of #345.

Current toolchain

Both workflows now build with the QuantEcon fork of the MyST CLI and the new theme, so the Netlify preview shows the latest content rendered by the latest tooling.

Piece What the branch uses Where it is set
MyST CLI QuantEcon/mystmd main, cloned and built from source with bun; quantecon/VERSION.yml is printed to the log so every build records the qe-vN fork state it ran with (currently v1.10.1 (qe-v8)) .github/workflows/ci.yml, .github/workflows/build-ipynb.yml
Theme quantecon-theme.mystmd v2.2.0, pinned to the release zip lectures/myst.yml (site.template)
Build command myst build --html --execute .github/workflows/ci.yml
Content merged up to date with main

The theme moves off the archived QuantEcon/quantecon-theme bundle (which the branch was still pulling as a floating main.zip) onto a pinned release of the new theme repo. Bump the vX.Y.Z in the site.template URL to take a newer theme; the fork stays on main deliberately, so the preview tracks the latest tooling.

The build was not actually using the fork

Worth calling out separately, because it had been green and invisible: until this update the HTML preview was built by jupyter book build, which cannot use the fork. jupyter-book 2.x is a thin Python shim — jupyter_book/__main__.py execs a compiled mystmd bundle vendored inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and exposes no hook to point it at another CLI. The globally installed fork was simply ignored, and only the build-ipynb workflow (which calls myst directly) ever exercised it.

The last run before the fix shows it plainly: the jupyter-book HTML build emitted 27 × 'output' unknown export output extension: exports/<page>.ipynb, because the per-page ipynb export is a fork feature (myst-to-ipynb, qe-v1) that upstream does not know about. The same commit built by the fork's myst build --ipynb emitted none and exported 23 clean notebooks. The HTML build now runs myst build --html --execute — the same engine and the same flags, since jupyter-book 2.x is mystmd, just our build of it.

Everything else in this update

  • Merged main. Brings in the new Polars lecture, the numba rewrite, the pandas_panel switch to remote CSVs, and the translation-sync workflows. The one conflict was ci.yml, where main and this branch have rewritten the same job for different builders — resolved in favour of this branch's job, keeping main's runner spec and action bumps.
  • myst.yml TOC re-synced with _toc.yml. It had drifted: autodiff and the new polars lecture were missing, workspace appeared twice, and one part title differed. The two files now list exactly the same 27 pages.
  • project.github corrected to lecture-python-programming — it pointed at lecture-python-programming.myst, which now only survives as a rename redirect. The theme derives the notebook-launch repo from this value plus a .notebooks suffix, so a stale name means broken launch buttons.
  • JAX is now installed in the build. Previous preview builds were rendering ModuleNotFoundError cells for jax_intro, autodiff and numpy_vs_numba_vs_jax, because this job installed only myst_requirements.txt while the jupyter-book 1.x job on main installs jax[cuda13] separately. Same install, same runner family.
  • CI actions moved to current majors and Node 20 → 24 (the theme's .nvmrc; it runs as a Remix server during myst build --html). This also clears the Node 20 runtime deprecation warnings. The unused configure-pages step is gone — this job deploys to Netlify, not Pages.
  • Removed committed execution artifacts (foo.py, newfile.txt, numbers.txt, output.txt, output2.txt, test_table.csv, us_cities.txt). All are written by %%file / %%writefile cells in the lectures themselves, all are matched by .gitignore, and none exist on main.

Still open

  • Nothing fails the build on a notebook execution error. Execution errors are reported and the build carries on, which is how the JAX failures above survived green builds. main gained a gate for the jupyter-book 1.x path in CI: gate the first jb build so notebook errors cannot pass green #588; the fork's CLI has myst build --strict for this, but it should be turned on only once the remaining build diagnostics are triaged, or it will fail on unrelated warnings.
  • Launch-button URLs are left on the theme defaults (<repo>.notebooks, branch main, notebooks at the repo root), matching the layout of lecture-python-programming.notebooks. Worth clicking through in the preview to confirm the page location resolves as expected.

@github-actions

github-actions Bot commented Jul 23, 2025

Copy link
Copy Markdown

@github-actions
github-actions Bot temporarily deployed to pull request July 23, 2025 01:19 Inactive
@kp992
kp992 requested a review from mmcky July 23, 2025 01:20
@mmcky

mmcky commented Jul 23, 2025

Copy link
Copy Markdown
Contributor

@kp992 thanks for opening this. Excellent work! It is exciting to see how close we are now.

From my review it looks like there a just a few issues and features we need before launch.

Bugs:

  1. LHS TOC menu flickers when using the Home page or a top level Part section. It is like it has an auto-stow feature that is triggering when pages are reloaded. It doesn't look great when pages are refreshed.
  2. Code block formatting has too much indentation for Solution blocks

Features:

  1. I see a bunch of new export notebooks being added to the yaml headers of each lecture. Do we need to do some development work on supporting the download notebook link in the theme?
  2. Is index support on the jb2 / mystmd roadmap?

- Add `markdown: commonmark` to all 24 lecture export configs so ipynb
  exports produce plain CommonMark markdown cells compatible with
  vanilla Jupyter Notebook, JupyterLab, and Google Colab
- Add build-ipynb.yml workflow that clones QuantEcon/mystmd@myst-to-ipynb,
  builds from source, exports all ipynb files, and audits for MyST leaks
- All 24 notebooks pass audit (0 MyST syntax leaks)

Uses QuantEcon/mystmd myst-to-ipynb branch which adds:
- CommonMark AST pre-transform (admonitions, math, figures, exercises, etc.)
- Identifier/label stripping to prevent (label)= prefixes
- Image directive stripping for plain ![alt](url) output
- Empty cell filtering and block marker removal
# Conflicts:
#	.github/workflows/ci.yml
@github-actions
github-actions Bot temporarily deployed to pull request May 14, 2026 05:44 Inactive
@mmcky

mmcky commented May 14, 2026

Copy link
Copy Markdown
Contributor
  • check if theme regression is occurring. Clicking on a chapter briefly shows a malformed webpage that disappears to the LEFT. NEEDS REVIEW.

mmcky and others added 3 commits July 31, 2026 16:08
Brings the branch up to date with main: the new Polars lecture, the numba
lecture rewrite, the pandas_panel switch to remote CSVs, the translation
sync workflows, and the jupyter-book 1.x CI changes.

The only conflict was .github/workflows/ci.yml, where main and this branch
have rewritten the same job for different builders. Resolved in favour of
this branch's jupyter-book 2.0 job, adopting main's runner spec
(volume=80gb/spot=false) and action bumps (checkout@v7,
upload-artifact@v7, actions-netlify@v4) on top of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oling

Theme: move site.template from the archived QuantEcon/quantecon-theme bundle
(main.zip) to a pinned quantecon-theme.mystmd release, v2.2.0 — the current
release, which carries the fancy-list rendering that pairs with the fork's
fancy-lists parser (qe-v8). Pinned rather than floating so a theme release
cannot change the build under us; bump the version in the URL to take a
newer one.

myst.yml also gets:

- project.github corrected to lecture-python-programming (it pointed at
  lecture-python-programming.myst, which only survives as a rename
  redirect). The theme derives the notebook-launch repo from this value
  plus a ".notebooks" suffix, so a stale name means broken launch buttons.
- toc brought back in line with _toc.yml: adds autodiff and the new polars
  lecture, drops the duplicate workspace entry from "Introduction to
  Python", and renames "The Scientific Libraries" to "Foundations of
  Scientific Computing".

CI: Node 20 -> 24 (the theme's .nvmrc; it runs as a Remix server during
`myst build --html`) and the remaining actions to current majors, which
also clears the Node 20 runtime deprecation warnings. Adds the JAX install
the jupyter-book 1.x workflow on main already does — without it the GPU
lectures execute to ModuleNotFoundError cells in the preview, which the
build does not fail on. Drops the unused configure-pages step; this job
deploys to Netlify, not Pages.

mystmd itself already comes from the QuantEcon fork's main branch, built
from source with bun, with quantecon/VERSION.yml logged for traceability.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
foo.py, newfile.txt, numbers.txt, output.txt, output2.txt, test_table.csv
and us_cities.txt are all written by the lectures themselves during
execution (%%file / %%writefile cells in getting_started, python_essentials,
python_advanced_features and debugging), and every one of them is matched
by .gitignore. They were picked up by the initial mystmd setup commit and
do not exist on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to pull request July 31, 2026 06:15 Inactive
The preview build was not using QuantEcon/mystmd at all. jupyter-book 2.x
is a thin Python shim: jupyter_book/__main__.py execs a compiled mystmd
bundle vendored inside its own wheel (jupyter_book/dist/jupyter-book.cjs)
and exposes no way to point it at a different CLI. So `jupyter book build`
ignored the fork this workflow installs globally and built the site with
upstream mystmd, while only the build-ipynb workflow (which calls `myst`
directly) ever exercised the fork.

The last run shows it plainly: the jupyter-book HTML build emitted 27 x
"'output' unknown export output extension: exports/<page>.ipynb", because
the per-page ipynb export is a fork feature (myst-to-ipynb, qe-v1) that
upstream does not know. The same commit built by the fork's `myst build
--ipynb` emitted none and exported 23 clean notebooks.

`myst build --html --execute` is the same engine and the same flags —
jupyter-book 2.x is mystmd — just our build of it. Dropping the
jupyter-book install also removes a second, conflicting mystmd from the
image. Workflow renamed to match what it now runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mmcky

mmcky commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Refreshed against the current toolchain

Branch is up to date with main and now builds with the QuantEcon fork of the MyST CLI and the new theme repo. The PR description has been rewritten to describe the resulting setup.

Piece Before Now
MyST CLI (HTML build) upstream mystmd, vendored inside the jupyter-book wheel QuantEcon/mystmd mainv1.10.1 (qe-v8)
Theme QuantEcon/quantecon-theme floating main.zip (repo archived) quantecon-theme.mystmd v2.2.0, pinned release zip
Content 2 months behind main merged, conflict resolved — PR is mergeable again

The HTML build was never using the fork

This is the part worth reading. jupyter book build cannot use our fork, and had been quietly building the preview with upstream mystmd this whole time. jupyter-book 2.x is a thin Python shim: jupyter_book/__main__.py execs a compiled mystmd bundle vendored inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and exposes no hook to point it at a different CLI. The npm install -g of the fork in this workflow was installing a CLI that the build step then ignored. Only build-ipynb.yml, which calls myst directly, was ever exercising the fork.

The evidence is in the last run before the fix — the jupyter-book HTML build emitted 27 × 'output' unknown export output extension: exports/<page>.ipynb, since the per-page ipynb export is a fork feature (myst-to-ipynb, qe-v1) that upstream doesn't know about. The same commit built by the fork's myst build --ipynb emitted none and exported 23 clean notebooks.

The HTML build now runs myst build --html --execute with the fork's CLI, and the pip install jupyter-book step is gone — it was only putting a second, conflicting mystmd on the image. Same engine and same flags either way; jupyter-book 2.x is mystmd, this is just our build of it.

Three other things the review turned up

  • The JAX lectures were rendering ModuleNotFoundError cells. This job installed only myst_requirements.txt, while the jupyter-book 1.x job on main installs jax[cuda13] separately, so jax_intro, autodiff and numpy_vs_numba_vs_jax had been executing to import errors through green builds. Same install added here.
  • myst.yml's TOC had drifted from _toc.ymlautodiff and the new polars lecture missing, workspace listed twice, one part title stale. The two files now list exactly the same 27 pages.
  • project.github pointed at lecture-python-programming.myst, which only survives as a rename redirect. The theme derives the notebook-launch repo from that value plus a .notebooks suffix, so a stale name means broken launch buttons. Corrected.

Also: Node 20 → 24 (the theme's .nvmrc), remaining actions to current majors (clears the Node 20 removal deprecation), the unused configure-pages step dropped since this job deploys to Netlify rather than Pages, and seven committed execution artifacts removed (foo.py, us_cities.txt, test_table.csv and friends — all written by %%file cells in the lectures, all gitignored, none present on main).

Left for a decision

Nothing fails the build on a notebook execution error — that is exactly how the JAX breakage stayed invisible. main gained a gate for the jupyter-book 1.x path in #588; the fork's CLI has myst build --strict for the same job here, but it is worth turning on only after the remaining build diagnostics are triaged, otherwise it will fail on unrelated warnings.

@mmcky

mmcky commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

This is supporting @DrDrij review of quantecon-theme.mystmd

mmcky and others added 2 commits August 3, 2026 19:28
The GitHub docs URL for "About pull requests" now 301-redirects to
https://docs.github.com/en/pull-requests/reference/pull-requests

Point directly at the destination so the weekly link checker stops
flagging it as a redirect.

Reported by the link checker in #587

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mmcky mmcky changed the title [Experimental] Use jupyter-book 2.0 for building the lectures [Experimental] Build the lectures with QuantEcon/mystmd instead of jupyter-book Aug 3, 2026
@mmcky

mmcky commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Why this branch builds with QuantEcon/mystmd rather than jupyter-book

Retitled to reflect what the build chain actually does. The short version: jupyter-book cannot be pointed at our fork, so building through it silently discards every QuantEcon feature we depend on.

jupyter-book ignores the fork, even when the fork is installed

jupyter-book 2.x is a Python shim that execs a mystmd bundle vendored inside its own wheel (jupyter_book/dist/jupyter-book.cjs). Installing QuantEcon/mystmd globally with npm afterwards puts our myst on PATH, but jupyter-book never calls it.

This is not a theoretical concern — it is visible in our own CI history. In the last run that built with jupyter book build (4314bc19, 2026-07-31), the Verify mystmd version step reported v1.10.1 (qe-v8), confirming the fork was installed and first on PATH. The build in the very next step then emitted 23 errors of the form ⛔️ unknown export output extension: exports/<lecture>.ipynb — one per lecture declaring an ipynb export. That export format is a fork-only capability, so those errors prove the build was running upstream mystmd, not ours. Under myst build with the fork, the same 23 exports produce zero errors.

The Verify mystmd version step is therefore misleading on its own: it verifies the standalone binary, not the engine jupyter-book actually execs. It reports a reassuring qe-v8 while the build beside it uses something else.

What we would lose

The fork is at qe-v8 on upstream_base: 1.9.1 and carries thirteen QuantEcon features, most of them load-bearing for a book-style render: CommonMark ipynb export with image attachments, book-style numbering, Roman-numbered part dividers with a continuous chapter counter, section-level scope for proof:* / figure / equation numbering, the full title→heading chain per section, shared counters across proof kinds, Pandoc-style fancy ordered lists, citation scanning that respects inline code spans, numbering.code honouring the book prefix, and amsthm-convention unnumbered proofs. None of these exist upstream, so none of them survive a jupyter book build.

What is and is not migrated

workflow engine
ci.yml (PR preview) myst build --html --execute, QuantEcon/mystmd
build-ipynb.yml myst build --html / --ipynb, QuantEcon/mystmd
publish.yml (production deploy) still jupyter-book 1.x jb build — untouched from main
cache.yml, execution-*.yml, environment.yml still jupyter-book 1.x — untouched from main

Migrating the production chain is deliberately out of scope here and belongs in its own PR: it covers the PDF build (currently jb build --builder pdflatex, with a typst book-pdf export stubbed in myst.yml), the notebook downloads (currently the sphinx-tojupyter custom builder, already prototyped as myst build --ipynb in build-ipynb.yml), the weekly execution cache, and the Anaconda execution tests.

Known issue on this branch

Cold-cache builds currently fail. The execution cache key is hashFiles('lectures/**/*.md'), so editing any single lecture invalidates all ~25 notebooks and forces a full re-execution; jax_intro.md's kernel is then killed (Connection lostKernel: restartingUnhandled error, exit 1) with no Python traceback.

This is not caused by mixing build systems — ci.yml installs no jupyter-book at all, and publish.yml / cache.yml do not run on pull requests. Comparing the last green cold run against the failures: same JAX 0.11.0, same 25 notebooks executed cold, same peak concurrency of 8, near-identical overlapping peers. The only difference is the engine — jax_intro.md executed in 25 s under jupyter-book's vendored mystmd and dies at 66 s under the fork's CLI. No timeout explanation survives either, since numpy.md took 99 s in the green run and built fine.

Two things worth doing before diagnosing further: pin the fork clone to the qe-v8 tag rather than --branch main, since the engine is currently a moving target between runs of the same commit; and capture free -m, nvidia-smi and dmesg -T | grep -i "killed process" in an if: always() step, because a SIGKILL with no traceback is consistent with the host OOM killer but that remains unconfirmed.

mmcky and others added 2 commits August 3, 2026 20:38
Diagnostics for the jax_intro.md kernel death on cold-cache runs
(kernel dies with no Python traceback ~66s into execution; numpy.md
is collateral). Instruments only -- execution itself is unchanged
(default parallelism, no XLA env) so the failure reproduces faithfully:

* Pin the QuantEcon/mystmd clone to the exact main SHA the failing
  runs used (main has not moved since 2026-06-12). A branch name here
  makes the engine a moving target between runs of the same commit.
* Detached 5s sampler logging host RAM, top-RSS processes and GPU
  memory for the life of the job.
* always() step reading the sampler log back plus dmesg, to confirm
  or rule out the host OOM killer -- a SIGKILL with no traceback is
  consistent with it but unproven.

The cache key only hashes lectures/**/*.md, so this ci.yml-only commit
reuses the failing runs' key; no saved cache exists under it (the
failed jobs saved none), guaranteeing the cold execution path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the cold-cache kernel death diagnosed in the instrumented run
(30806359596). The three JAX lectures execute concurrently on the
g4dn runner's single T4; whichever kernel touches the GPU first
preallocates JAX's default 75% (11.3GiB of 15.3GiB). The diagnostics
showed jax_intro's kernel losing that race, clamping to the ~3.4GiB
remainder, and aborting ~40s later when a cell outgrew the pool --
an XLA fatal with no Python traceback, no OOM-killer or segfault
records in dmesg, and GPU usage pinned at 3427MiB until the death.

The jupyter-book-era green cold run won the same race by seconds --
the engine swap changed execution phasing, not semantics -- and the
JB1 production build on main executes sequentially, which is why
neither ever hit this.

XLA_PYTHON_CLIENT_PREALLOCATE=false switches JAX to on-demand
allocation so the concurrent kernels share the GPU. The memory
monitor and dmesg diagnostics stay in place; this run doubles as
the fix's validation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hardening after the preallocation-race incident. The cache key now
hashes myst_requirements.txt and ci.yml itself alongside the
lectures, so any engine SHA bump, JAX bump, XLA env change or
requirements change busts the cache and is validated by a cold
build automatically -- hashing only the lectures is how the
b9cc649 engine swap shipped green without executing a notebook.

JAX is pinned to 0.11.0 (the version the 2026-08-03 diagnosis and
fix were validated against); a bump is now a deliberate ci.yml edit
that itself forces cold validation.

Also rewords the diagnosis-era comments: the memory monitor and
dmesg read-back stay as standing telemetry.

This commit changes the cache key, so its own CI run is cold --
which doubles as a second cold-green validation of the
XLA_PYTHON_CLIENT_PREALLOCATE fix under the pinned engine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mmcky

mmcky commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Cold-cache execution failure: diagnosed, fixed, validated

Closing out the "known issue" from my earlier comment. The kernel death was not the engine and not host memory — it was a GPU preallocation race, confirmed by an instrumented run and fixed in two commits.

Diagnosis (instrumented run 30806359596)

A 5-second GPU/host-memory sampler ran beside the failing build. The GPU timeline told the whole story on one line: 0 → 11,305 MiB → 14,727 → 3,427 → 0 on the g4dn's 15,360 MiB T4.

The three JAX lectures (jax_intro, autodiff, numpy_vs_numba_vs_jax) execute concurrently (myst parallelism = cpus−1 = 7). The first JAX kernel to touch the GPU preallocated JAX's default 75% — 11,305 MiB. jax_intro's kernel lost that race, clamped to the ~3.4 GiB remainder, and aborted ~40 s later when a cell outgrew the pool — an XLA fatal in the kernel process, which is why there was no Python traceback. dmesg was clean: no OOM-killer records, no segfaults, so the host is exonerated. The only kernel event in the entire build was jax_intro's; the 11.3 GiB holder's exit was a clean post-build reap.

This also explains the confusing history: the jupyter-book-era cold run went green because jax_intro won the race that day (verified from its HTML artifact — JAX cos on 50M elements in 0.0022 s vs NumPy's 1.07 s, i.e. genuinely on GPU), and main never hits this because the JB1 build executes notebooks sequentially. The engine swap changed execution phasing, not semantics — which is why the failure appeared engine-correlated for three runs straight.

Fix and validation

commit change
8af1015 instrumentation + engine pinned to the fork's exact SHA (96ee78f) instead of --branch main
94a7e52 XLA_PYTHON_CLIENT_PREALLOCATE: 'false' on the build step — concurrent kernels allocate on demand
f541ccc hardening: cache key now hashes myst_requirements.txt + ci.yml alongside the lectures, JAX pinned to 0.11.0

Validation run 30806924581: fully cold (all 25 notebooks executed), zero kernel restarts, jax_intro built in 46 s (vs 45 s in the healthy jupyter-book-era cold run), peak GPU usage 4,217 MiB with no preallocation spike, dmesg clean. The f541ccc run is a second cold validation by construction, since changing the cache key forces one.

Structural fixes, so this class of bug can't ship green again

The engine swap originally shipped green because the execution cache key hashed only lectures/**/*.md — a warm cache replayed old outputs and the new engine never executed a notebook. The key now hashes the requirements and the workflow file itself, so any engine SHA bump, JAX bump, or XLA env change forces a cold, genuinely-executed build automatically. The memory sampler and dmesg read-back stay in the workflow as standing telemetry.

One upstream follow-up filed from this: myst reports a dead kernel as a bare Unhandled error with no notebook, cell, cause, or kernel stderr — QuantEcon/mystmd#77 asks for the diagnostics that would have made this a one-look failure instead of an instrumented investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants